Automatic structuring of text files 1

SUMMARY In many practical information retrieval situations, it is necessary to process heterogeneous text databases that vary greatly in scope and coverage, and deal with many different subjects. In such an environment it is important to provide flexible access to individual text pieces, and to structure the collection so that related text elements are identified and appropriately linked. Methods are described in this study for the automatic structuring of heterogeneous text collections, and the construction of browsing tools and access procedures that facilitate collection use. The proposed methods are illustrated by performing searches with a large automated encyclopedia. In conventional information retrieval environments, documents are accessed by constructing large, so-called inverted, indexes containing all the distinct text words (except for some throwaway words), together with lists of document references that identify the documents, or document excerpts, in which the given text words occur. Information is retrieved by formulating Boolean queries consisting of text words interrelated by Boolean operators, consulting the corresponding lists of document references in the index, and identifying all documents that contain the proper combination of query terms. The retrieval technology using inverted term indexes together with Boolean query formulations has the considerable advantage that the identification of documents containing the required query term combination is extremely rapid. In general the responses are available in a matter of seconds, even when the file contains several million documents. Moreover, the retrieval effectiveness may be relatively high because the documents considered for retrieval, corresponding to entries in the appropriate reference lists in the index, are known in advance to contain at least one of the required search terms. On the negative side, the conventional technology does not offer browsing capabilities, because the files are maintained in random rather than subject matter order. Occasionally users are turned off when errors in the Boolean query formulations produce questionable responses. For example, in answer to the query " Esau and Jacob " (the two biblical characters in the Old Testament), an automated encyclopedia may retrieve the encyclopedia article on